.com
Hosted by:
Unit testing expertise at your fingertips!
Home | Discuss | Lists

Manual Intervention

The book has now been published and the content of this chapter has likely changed substanstially.
Please see page 250 of xUnit Test Patterns for the latest information.

A test requires a person to perform some manual action each time it is run.

Symptoms

The person running the test must do something manually either before the test is run or part way through the test run otherwise the test fails. A person may need to verify the results of the test manually.

Impact

Automated tests are all about getting early feedback on problems introduced into the software. If the cost of getting that feedback is too high in the form of Manual Intervention, we likely won't run the tests very often and we won't get the feedback very often. If we don't get that feedback very often, we'll probably introduce lots of problems between test runs and that will lead to Frequent Debugging (page X) and High Test Maintenance Cost (page X).

Manual Intervention also makes it impractical to have a fully-automated Integration Build[SCM] and regression test process.

Causes

The causes of Manual Intervention are as varied as the kinds of things our software does or encounters. The following are some general categories of the kinds of things that require Manual Intervention. This list is by all means not exhaustive.

Cause: Manual Fixture Setup

Symptoms

A person has to set up the test environment manually before the automated tests can be run. This may take the form of configuring servers, starting server processes or running scripts to set up a Prebuilt Fixture (page X).

Root Cause

This is typically caused by a lack of attention to automating the fixture setup phase of the test. It may also be caused by excessive coupling between components in the system under test (SUT) that prevent us from testing a majority of the code in the system inside the development environment.

Possible Solution

When using a Prebuilt Fixture it is important not to modify anything in the fixture to ensure we don't have Unrepeatable Tests (see Erratic Test on page X). This is called an Immutable Shared Fixture (see Shared Fixture on page X).

Cause: Manual Result Verification

Symptoms

We can run the tests but they almost always pass even when we know that the SUT is not returning the correct results.

Root Cause

If the tests we write are not Self-Checking Test (see Goals of Test Automation on page X), we can be given a false sense of security because tests will only fail if an error/exception is thrown.

Possible Solution

We need to ensure our tests are all self-checking by including result verification logic such as calls to Assertion Methods (page X) with the Test Methods (page X).

Cause: Manual Event Injection

Symptoms

A person must intervene during test execution to perform some manual action before the test can proceed.

Root Cause

Many events in a SUT are hard to generate under program control. Examples include unplugging network cables, bringing down database connections and pressing buttons on a user interface.

Impact

If a person needs to do something manually, it both increases the effort to run the test and ensures that the test cannot be run unattended. This torpedoes any attempt to do a fully-automated build & test cycle.

Possible Solution

The best solution is to find ways to test the software that do not require a real person to do the manual actions. If the events are reported to the SUT through asynchronous events, we can have the Test Method invoke the SUT directly passing it a simulated event object. If the SUT experiences the situation as a synchronous response from some other part of the system, we can get control of the indirect inputs by replacing some part of the SUT with a Test Stub (page X) that simulates the circumstances to which we want to expose the SUT.

Further Reading

Refer to the chapter Using Test Doubles for a much more detailed description of how to get control of the indirect inputs of the SUT.

Page generated at Wed Feb 09 16:39:50 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved

All Categories
Introductory Narratives
Web Site Instructions
Code Refactorings
Database Patterns
DfT Patterns
External Patterns
Fixture Setup Patterns
Fixture Teardown Patterns
Front Matter
Glossary
Misc
References
Result Verification Patterns
Sidebars
Terminology
Test Double Patterns
Test Organization
Test Refactorings
Test Smells
Test Strategy
Tools
Value Patterns
XUnit Basics
xUnit Members
All "Test Smells"
Code Smells
--Obscure Test
--Conditional Test Logic
--Hard-to-Test Code
--Test Code Duplication
--Test Logic in Production
Behavior Smells
--Assertion Roulette
--Erratic Test
--Fragile Test
--Frequent Debugging
--Manual Intervention
----Manual Fixture Setup
----Manual Result Verification
----Manual Event Injection
--Slow Tests
Project Smells
--Buggy Tests
--Developers Not Writing Tests
--High Test Maintenance Cost
--Production Bugs